Optical sensor for imaging an object

ABSTRACT

The sensor comprises a structured light source (5, 6, 7) which is adjustable so as to interchange the positions of contrasting areas of the pattern it provides, a detector (1) which comprises an array of detector elements having dimensions matched to the pattern produced by the light source, an optical system (2, 8) for projecting a primary image of the light source pattern onto an object (3) that is to be sensed and for forming a secondary image on the detector (1) of the primary image thus formed on the object (3), positioning means (4) for moving at least part (2) of the optical system so as to vary the focussing of the primary image on the object (3) and processing means (12) for analyzing signals produced by the detector (1) in conjunction with information on the adjustment of the optical system (2, 8). The optical arrangement is `confocal` so that, when the primary image is in focus on the object (3), the secondary image on the detector (1) is also in focus. The processing means (12) is arranged to analyse the images received by the detector (1) with the contrasting areas thereof in the interchanged positions to determine which parts of the images are in focus and hence determine the range of the corresponding parts of the object being viewed.

TECHNICAL FIELD

This invention relates to an optical sensor and, more particularly, to asensor used to sense the range of one or more parts of an object so asto form an image thereof.

BACKGROUND ART

A variety of different optical sensors are available for providing animage of objects within the field of view of the sensor. One such systemknown as a `sweep focus ranger` uses a video camera with a single lensof very short depth of field to produce an image in which only a narrowinterval of range in object space is in focus at any given time. Byusing a computer-controlled servo drive, the lens is positioned (or`swept`) with great accuracy over a series of positions so as to viewdifferent range `slices` of an object. A three-dimensional image of theobject is then built up from these `slices`. The system detects whichparts of the object are in focus by analysing the detected signal forhigh frequency components which are caused by features, such as edges ortextured parts, which change rapidly across the scene. Because of this,the system is not suitable for imaging plain or smooth surfaces, such asa flat painted wall, which have no such features.

This limitation is common to all passive rangefinding techniques. Oneway to overcome the problem is to actively project a pattern of lightonto the target objects which can then be observed by the sensor. Ifthis pattern contains high spatial frequencies, then these features canbe used by the sensor to estimate the range of otherwise plain surfaces.A particularly elegant way of projecting such a pattern is described inU.S. Pat. No. 4,629,324 by Robotic Vision Systems Inc.

In this prior art, the sensor detects those parts of the target objectthat are in focus by analysing the image for features that match thespatial frequencies present in the projected pattern. Various analysistechniques such as convolution and synchronous detection are described.However, such methods are potentially time consuming. In an extension ofthese ideas a further patent by the same company U.S. Pat. No. 4,640,620describes a method which aims to overcome this problem by the use of aliquid crystal light valve device to convert the required high spatialfrequency components present in the image into an amplitude variationthat can be detected directly.

The present invention aims to provide a simpler solution to thisproblem.

DISCLOSURE OF INVENTION

According to one aspect of the present invention, there is provided anoptical sensor comprising: a structured light source for producing apattern of contrasting areas; a detector which comprises an array ofdetector elements having dimensions matched to the pattern produced bythe light source; an optical system for projecting a primary image ofthe light source onto an object that is to be sensed and for forming asecondary image on the detector of the primary image thus formed on theobject; adjustment means for adjusting at least part of the opticalsystem so as to vary the focussing of the primary image on the object,the arrangement being such that when the primary image is in focus onthe object, the secondary image on the detector is also in focus; andprocessing means for analysing signals produced by the detector inconjunction with information on the adjustment of the optical system,wherein the structured light source is adjustable so as to interchangethe positions of contrasting areas of the pattern produced by the lightsource and in that the processing means is arranged to analyse thesecondary images received by the detector elements with the contrastingareas in the interchanged positions to determine those parts of thesecondary images which are in focus on the detector and therebydetermine the range of corresponding parts of the object which are thusin focus.

According to another aspect of the invention, there is provided a methodof determining the range of at least part of an object being viewedusing an optical sensor comprising: a structured light source whichproduces a pattern of contrasting areas; a detector which comprises anarray of detector elements having dimensions matched to the patternproduced by the light source; an optical system which projects a primaryimage of the light source onto an object that is to be sensed and formsa secondary image on the detector of the primary image thus formed onthe object; adjustment means which adjusts at least part of the opticalsystem so as to vary the focussing of the primary image on the object,the arrangement being such that when the primary image is in focus onthe object, the secondary image on the detector is also in focus; andprocessing means which analyses signals produced by the detector inconjunction with information on the adjustment of the optical system,the method involving adjustment of the structured light source so as tointerchange the positions of contrasting areas of the pattern producedby the light source and the processing means being arranged to analysethe secondary images received by the detector elements with thecontrasting areas in the interchanged positions to determine those partsof the secondary images which are in focus on the detector and therebydetermine the range of corresponding parts of the object which are thusin focus.

The invention thus transforms analysis of the image sensed by thedetector into the temporal domain.

Preferred and optional features of the invention will be apparent fromthe following description and from the subsidiary claims of thespecification.

BRIEF DESCRIPTION OF DRAWINGS

The invention will now be further described, merely by way of example,with reference to the accompanying drawings, in which:

FIG. 1 illustrates the basic concept of an optical sensor such as thatdescribed in U.S. Pat. No. 4,629,324 and shows the main components andthe optical pathways between them;

FIG. 2 shows a box diagram of an optical sensor of the type shown inFIG. 1 with a particular control unit and signal processing arrangementaccording to one embodiment of the present invention;

FIG. 3 illustrates how an image of an object being viewed by the type ofsystem shown in FIGS. 1 and 2 can be built up and displayed; FIGS. 4, 5and 6 show alternative optical arrangements which may be used in thesensor;

FIG. 7 shows an alternative form of beam splitter which may be used inthe sensor;

FIGS. 8(A) and 8(B) show a further embodiment of a sensor according tothe invention;

FIGS. 9(A) and 9(B) show another embodiment of a sensor according to theinvention; and

FIG. 10 shows a box diagram of a control unit and a signal processorwhich may be used with the embodiment shown in FIG. 9.

BEST MODE OF CARRYING OUT THE INVENTION

The optical sensor described herein combines the concepts of a `sweepfocus ranger` of the type described above with the concept of an activeconfocal light source as described in U.S. Pat. No. 4,629,324 togetherwith means to transform the depth information analysis into the temporaldomain.

The term `confocal` is used in this specification to describe an opticalsystem arranged so that two images formed by the system are in focus atthe same time. In most cases, this means that if the relevant opticalpaths are `unfolded`, the respective images or objects coincide witheach other.

The basic concept of a sensor such as that described in U.S. Pat. No.4,629,324 is illustrated in FIG. 1. The sensor comprises a detector 1and a lens system 2 for focussing an image of an object 3 which is to beviewed onto the detector 1. Positioning means 4 are provided to adjustthe position of the lens system 2 to focus different `slices` of theobject 3 on the detector 1. Thus far, the sensor corresponds to aconventional `sweep focus sensor`. As described in U.S. Pat. No.4,629,324, the sensor is also provided with a structured light sourcecomprising a lamp 5, a projector lens 6 and a grid or spatial filter 7together with a beam-splitter 8 which enables an image of the filter 7to be projected through the lens system 2 onto the object 3. The filter7 and detector 1 are accurately positioned so that when a primary imageof the filter 7 is focussed by the lens system 2 onto the object 3, asecondary image of the primary image formed on the object 3 is alsofocussed by the lens system 2 onto the detector 1. This is achieved bypositioning the detector 1 and filter 7 equidistant from the beamsplitter 8 so the system is `confocal`. An absorber 9 is also providedto help ensure that the portion of the beam from the filter 7 whichpasses through the beam splitter 8 is absorbed and is not reflected ontothe detector 1.

The lens system 2 has a wide aperture with a very small depth of focusand in order to form an image of the object 3 the lens system 2 issuccessively positioned at hundreds of discrete, precalculated positionsand the images received by the detector 1 for each position of the lenssystem 2 are analysed to detect those parts of the image which are infocus. When the images of the grid 7 on the object 3 and on the detector1 are in focus, the distance between the lens system 2 and the parts ofthe object 3 which are in focus at that time can be calculated using thestandard lens equation; which for a simple lens is:

    1/f=1/U+1/V

where f is the focal length of the lens system 2, U is the objectdistance (i.e. the distance between the lens system 2 and the in focusparts of the object 3) and V is the image distance (i.e. the distancebetween the lens system 2 and the detector 1).

To enable areas of object which are in-focus to be detected, the imagesformed on the object 3 and detector 1 are preferably in the form of auniform structured pattern with a high spatial frequency, i.e. having asmall repeat distance, comprising contrasting areas, such as a series oflight and dark bands. Such patterns can, for example, be provided byslots or square cut-outs in the filter 7. For such patterns, those partsof the image which are in focus on the object 3 being viewed produce acorresponding image on the detector 1 of light and dark bands whereasthe out of focus parts of the image rapidly `break-up` and so produce amore uniform illumination of the detector 1 which is also significantlyless bright than the light areas of the parts of the image which are infocus.

The structured pattern should preferably have as high a spatialfrequency as possible, although this is limited by the resolution of thelens system 2 and the size of the detector array, as the depthresolution is proportional to this spatial frequency.

FIG. 2 is a box diagram of a sensor of the type shown in FIG. 1 togetherwith control and processing units as used in an embodiment of theinvention to be described. The light source 5 of the sensor iscontrolled by a projector lamp control unit 9, a piezo-electric gridcontrol unit 10 is provided for moving the grid or filter 7 (for reasonsto be discussed later) and adjustment of the lens system 2 is controlledby a sweep lens control unit 11. The control units 9, 10 and 11,together with the output of the CCD detector 1, are connected to acomputer 12 provided with an image processor, frame grabber, framestores (computer memory corresponding to the pixel array of thedetector 1) and a digital signal processor (DSP) 13 for processing thesignals received and providing appropriate instructions to the controlunits. The output of the signal processor may then be displayed on amonitor 14.

In the sensor described herein, the detector 1 comprises an array ofdetector elements or pixels such as charged coupled devices (CCD's) orcharged injection devices (CID's). Such detectors are preferred overother TV type sensors because the precise nature of the detectorgeometry makes it possible to align the unfolded optical path of thestructured image with the individual pixels of the detector array.

The grid or spatial filter 7 consists of a uniform structured patternwith a high spatial frequency i.e. having a small repeat distance,comprising contrasting areas, such as a series of light and dark bandsor a chequer-board pattern. The spatial frequency of the grid pattern 7should be matched to the pixel dimensions of the detector 1, i.e. therepeat distance of the pattern should be n pixels wide, where n is asmall even integer e.g. two (which is the preferred repeat distance).The value of n will be limited by the resolution of the lens system 2.

If a chequer-board type pattern is used, the detector should be matchedwith the pattern in both dimensions, so for optimum resolution it shouldcomprise an array of square rather than rectangular pixels as the lenssystem 2 will have the same resolving power in both the x and ydimensions. Also, the use of square pixels simplifies the analysis ofdata received by the detector.

The light and dark portion of the pattern should also be complementaryso that added together they give a uniform intensity. The choice ofpattern will depend on the resolution characteristics of the opticalsystem used but should be chosen such that the pattern `breaks up` asquickly and cleanly as possible as it goes out of focus. The patternneed not have a simple light/dark or clear/opaque structure. It couldalso comprise a pattern having smoothly changing (e.g. sinusoidal)opacity. This would tend to reduce the higher frequency components ofthe pattern so that it `breaks up` more smoothly as it moves out offocus.

In order to avoid edge effects in the image formed on the object 3 ordetector 1, it is preferable for the grid pattern to comprise morerepeat patterns than the CCd detector, i.e. for the image formed to belarger than the detector array 1.

One possible form of grid pattern 7 is that known as a `Ronchi rulingresolution target` which comprises a pattern of parallel stripes whichare alternatively clear and opaque. A good quality lens system 2 (suchas that used in a camera) will typically be able to resolve a patternhaving between 50 and 125 lines/mm in the image plane depending upon itsaperture, and special lenses (such as those used in aerial-photography)would be even better. The resolution can also be further improved bylimiting the wavelength band used, e.g. by using a laser or sodium lampto illuminate the grid pattern 7. A typical CCD detector has pixelsspaced at about 20 microns square so with two pixel widths, i.e. 40microns, corresponding to the pattern repeat distance, this sets aresolution limit of 25 lines/mm. Finer devices are likely to becomeavailable in the future.

The number of repeats of the structured pattern across the area of theobject being viewed should preferably be as high as possible for thelens system and detector array used. With a typical detector array ofthe type described above comprising a matrix of 512×512 pixels, themaximum number of repeats would be 256 across the image. Considerablyhigher repeat numbers can be achieved using a linear detector array(described further below).

The lower limit on the spatial frequency of the grid pattern is set bythe minimum depth resolution required. As the grid pattern becomescoarser, the number of repeats of the pattern across the image isreduced and the quality of the image produced by the sensor is degraded.An image with less than, say 40 pattern repeats across it is clearlygoing to provide only very crude information about the object beingsensed.

In the sensor described herein, the high spatial frequency component ofthe light source is converted into the temporal domain. As indicated inFIG. 2, positioning means may be provided for moving the grid pattern 7.This is used to move the grid 7 in its own plane by one half of thepattern repeat distance between each frame grab. This movement istypically very small, e.g. around 20 microns (where the pattern repeatsevery 2 pixels), and is preferably carried out using piezo-electricpositioning means. The effect of this movement is that in one positionof the grid the unfolded optical path of the grid pattern overlaps thedetector pixels in such a way that half the pixels correspond to lightareas and half to dark and when the grid is moved by one half repeatdistance then the opposite situation exists for each pixel (the patternof light and dark areas on the array of pixels corresponding to thepattern of the structured light source). Each pixel of the detector 1 isthus mapped to an area of the pattern produced by the light source whichalternates between light and dark as the structured light source ismoved. Pairs of images may then be captured in the first and secondframe stores with the grid 7 in the two positions and the intensities(i1 and i2) of corresponding pixels in the two frame stores determinedso the following functions can be produced for each pixel: ##EQU1##

The sum of the intensities i1 and i2 is a measure of the brightness ofthat pixel and the difference a measure of the depth of modulation.Dividing the difference signal by the brightness gives a normalizedmeasure of the high pass component, which is a measure of how `in-focus`that pixel is. The sign of the term "i1-i2" will, of course, alternatefrom pixel to pixel depending whether i1 or i2 corresponds to a lightarea of the grid pattern.

Those parts of the image which are in focus on the object 3 and on thedetector 1, produce a pattern of light and dark areas which alternate asthe grid 7 is moved. One of the signals i2 and i2 will therefore be high(for a bright area) and one will be low (for a dark area). In contrast,for parts of the image which are not in focus on the detector 1, theintensities i1 and i2 will be similar to each other and lower than theintensity of an in focus bright area.

If the background illumination is significant, a correction can beapplied by capturing an image with the light source switched off in athird frame store. The background intensity "i3" for each pixel can thenbe subtracted from the values i1 and i2 used in the above equations.

In order to avoid spurious, noisy data, it is desirable to impose aminimum threshold value on the signal `I`. Where `I` is very low, forexample when looking at a black object or when the object is verydistant, then that sample should be ignored (e.g. by setting M to zeroor some such value).

The process of constructing a complete 3D surface map of an object beingviewed may thus proceed as follows:

1) Clear the 3D surface model map file.

2) Set the lens sweep to the starting position.

3) For each lens sweep position three frames are captured into threeframe stores as follows:

1=Illumination on, grid pattern in position 1

2=Illumination on, grid pattern in position 2

3=No Illumination

4) The DSP is now used to perform the following functions on the rawimage data: for each pixel (or group of pixels) the functions I and Mdescribed above are constructed.

5) The background signal is then subtracted:

i1:=i1-i3

i2:=i2-i3

6) Construct mean intensity I into frame store 3:

I=i3:=i1+i2

7) Construct modulation depth M if I exceeds threshold value into framestore 1: ##EQU2## 8) The function M can be displayed from frame store 1on the monitor (this corresponds to the in-focus contours which areshown bright against a dark background).

9) Clean up this `M` data:

a) Check for pixel to pixel continuity.

b) Look for the local maxima positions, interpolating to sub-pixelpositions, then convert to corresponding object coordinates.

c) Construct data chains describing the in-focus contour positions.

d) Compare these contours with the adjacent sweep position contours.

e) Add this sweep position contours to the 3D surface model map.

10) Proceed to next lens sweep position and repeat the above stagesuntil a complete surface model map is constructed.

The intensity information signal `I`, which corresponds to theinformation obtained by a conventional sweep focus ranger as the averageof i1 and i2 is effectively a uniform illumination signal (i.e. withouta structured light source), is also helpful in interpreting the data.Displaying the signal `I` on a monitor gives a normal TV style image ofthe scene except for the very small depth of focus inherent in thesystem. If desired, changing patterns of intensity found using standardedge detector methods can therefore be used to provide information onin-focus edges and textured areas of the object and this information canbe added to that obtained from the `M` signal (which is provided by theuse of a structured light source).

The technique of temporal modulation has the advantage that as eachpixel in the image is analysed independently, edges or textures presenton the object do not interfere with the depth measurement (this may notbe the case for techniques based upon spatial modulation). The techniquecan also be used with a chequer-board grid pattern instead of verticalor horizontal bands. Such a pattern would probably `break-up` moreeffectively than the bands as the lens system 2 moves out of focus andwould therefore be the preferred choice. It should be noted that thesensors described above use a structured light source of relatively highspatial frequency to enable the detector to detect accurately when theimage is in focus and when it `breaks-up` as it goes out of focus. Thisis in contrast with some known range finding systems which rely ondetecting a shift between two halves of the image as it goes out offocus.

The calculations described above may be performed pixel by pixel insoftware or by using specialist digital signal processor (DSP) hardwareat video frame rates. A display of the function M on a CRT would havethe appearance of a dark screen with bright contours corresponding tothe "in-focus" components of the object being viewed. As the focus ofthe lens system is swept, so these contours would move to show thein-focus parts of the object. From an analysis of these contours a3-dimensional map of the object can be constructed.

A piezo-electric positioner can also be used on the detector 1 toincrease the effective resolution in both the spatial and depthdimensions. To do this, the detector array is moved so as to effectivelyprovide a smaller pixel dimension (a technique used in some highdefinition CCD cameras) and thus allow a finer grid pattern to be used.

The use of piezo-electric devices for the fine positioning of an articleis well known, e.g. in the high definition CCD cameras mentioned aboveand in scanning tunnelling microscopes, are capable of very accuratelypositioning an article even down to atomic dimensions. As an alternativeto using a piezo-electric positioner, other devices could be used suchas a loudspeaker voice coil positioning mechanism.

Instead of moving part of the light source it is also possible tomodulate the light source directly, e.g. by using:

1) a liquid crystal display (LCD) (e.g. of the type which can be usedwith some personal computers to project images onto an overheadprojector)

2) a small cathode ray tube (CRT) to write the pattern directly

3) magneto optical modulation (i.e. the Faraday Effect) together with apolarized light.

4) a purpose made, interlaced, fibre optic light guide bundle with twoalternating light sources.

FIG. 3 illustrates how the lens system 2 is adjusted to focus onsuccessive planes of the object 3, and a 3-dimensional electronic image,or range map, of the object is built up from these `slices` in thecomputer memory. The contours of the object 3 which are in focus for anyparticular position of the lens system 2 can be displayed on the monitor14. Other images of the 3-dimensional model built up in this way canalso be displayed using well known image display and processingtechniques.

The optical sensor described above is an `active` system (i.e. uses alight source to illuminate the object being viewed) rather than passive(i.e. relying on ambient light to illuminate the object). As it projectsa pattern onto the object being viewed it is able to sense plain,un-textured surfaces, such as painted walls, floors, skin, etc., and isnot restricted to edge or textured features like a conventional sweepfocus ranger. The use of a pattern which is projected onto the object tobe viewed and which is of a form which can be easily analysed todetermine those parts which are in focus, thus provides significantadvantages over a conventional `sweep focus ranger`. In addition, sincethe outgoing projected beam is subject to the same focussing sweep asthe incoming beam sensed by the detector 1, the projected pattern isonly focussed on those parts of the object which are themselves infocus. This improves on the resolving capability of the conventionalsweep focus ranger by effectively `doubling up` the focussing actionusing both the detector 1 and the light source. The symmetry of theoptical system means that when the object is in focus, the spatialfrequency of the signal formed at the detector 1 will exactly equal thatof the grid pattern 7.

The scale of measurements over which the type of sensor described abovecan be used ranges from the large, e.g. a few meters, down to the small,e.g. a few millimeters. It should also be possible to apply the samemethod to the very small, so effectively giving a 3D microscope, usingrelatively simple and inexpensive equipment.

In general, the technology is easier to apply at the smaller rather thanthe larger scale. This is because for sensors working on the smallscale, the depth resolution is comparable with the spatial resolution,whereas on the larger scale the depth resolution falls off (which is tobe expected as the effective triangulation angle of the lens system isreduced). Also, being an active system, the illumination required willincrease as the square of the distance between the sensor and the objectbeing viewed. Nevertheless, the system is well suited to use as a robotsensor covering a range up to several meters.

One of the problems with conventional microscopes is the "fuzzyness"that results from the very small depth of focus. To try to overcomethis, CCD cameras have been attached to microscopes to capture imagesdirectly and software packages are now available which can process aseries of captured images and perform a de-convolution operation toremove the fuzzyness. In this way images of a clarity comparable withthose obtainable with a very much more expensive confocal scanningmicroscope are possible. The method described above of capturing anddifferencing pairs of frames, also enables clear cross-sectional imagesto be formed. This is in many ways easier than the software approach asthe data contains height information in a much more accessible form. Byscanning over height as described (by moving the lens system 2) a 3Dimage of the sample can be built up. In the case of biological sampleswhich are of a translucent nature, it may no longer be appropriate toform a simple surface profile. Instead, it may be more appropriate toform a full 3D "optical density" model to reflect this quality of thesample.

Another common microscopy technique that can be adapted for use with thesensor described above is fluorescence. This involves staining thesample with a fluorescent dye so that when illuminated with light of aparticular wavelength, e.g. UV light, it will emit light of some otherwavelength, say yellow. In this way it is possible to see which part ofa sample has taken up the dye. If a light source of the appropriatewavelength is used and the detector arranged to respond only to theresulting fluorescence by the use of an appropriate filter, a 3Dfluorescence model of the sample can be built up.

It will be appreciated that in such applications, the lens system 2 actsas the microscope objective lens.

Various optical arrangements can be used in the type of sensor describedabove and some of these are illustrated in FIGS. 4, 5 and 6.

FIG. 4 shows a "side-by-side" arrangement in which separate lens systems2A and 2B are used to focus a primary image of the grid 7 on the object3 and to focus the secondary image thereof on the CCD detector 1. Thesystem is again arranged so that when the primary image is in focus onthe object 3, the secondary image on the detector is also in focus. Thisis achieved by making the lens systems 2A and 2B identical andpositioning them side by side or one above the other (depending whethera horizontal or vertical pattern is used) so they are equi-distant fromthe projector grid 7 and detector 1, respectively. Movement of the twolens systems 2A and 2B would also be matched during the sweeping action.The two optical systems in this case are effectively combined by overlapon the object 3 of the area illuminated by the projector grid 7 and thearea sensed by the detector 1.

FIG. 5 shows an arrangement in which separate optical systems arecombined using a mirror (or prism) 15 and a beam-splitter 16. Again,care is taken to ensure that the two lens systems 2A and 2B areidentical are accurately positioned and their movements matched, toensure the system is `confocal`.

FIG. 6 shows part of an arrangement corresponding to that of FIG. 1 inwhich an intermediate imaging stage is inserted in the detector optics.The secondary image is first focussed on a translucent screen 17, e.g.of ground glass, by the optical system 2 and then focussed onto thedetector 1 by a detector lens 18. The arrangement shown in FIG. 6 isparticularly suited to long range sensors (e.g. more than 2 m) where itis desirable to use a larger lens (to give better depth resolution) thanwould normally be applicable for use with a CCD detector which istypically about 1 cm square.

If a structured light source having a pattern within a repeat distancegreater than two pixels (i.e. n>2), the preferred solution is to use anadditional imaging stage as in FIG. 6 to effectively reduce n to two.

However, larger CCD arrays are likely to become more readily availablein the near future.

When the sensor is designed to work on the very small scale, it iseasier to sweep not just the lens system 2 but the entire sensorrelative to the object being viewed (as in a conventional microscope).As the lens to detector distance remains constant, the field of viewangle and the magnification remain constant so each pixel of thedetector 1 corresponds to a fixed x,y location in the object plane.Analysis of the image formed on the detector 1 is thus greatlysimplified.

When the system is used as a microscope, it is preferable to use amirror type of lens system, such as a Cassegrain mirror type microscopeobjective, as this offers a much larger aperture, and hence better depthresolution, than a conventional lens system.

In each of the arrangements described above, the positions of the lightsource and detector 1 may also be interchanged if desired.

The beam splitter 8 may be a simple half-silvered mirror of conventionaldesign. However, a prism type of beam splitter, as shown in FIG. 7, ispreferred to help reduce the amount of stray light reaching thedetector 1. As shown in FIG. 7, total internal reflection prevents lightfrom the grid pattern 7 impinging directly on the detector 1. Polarizingbeam splitters combined with polarizing filters may also be used toreduce further the problems caused by stray light reaching the detector1.

The embodiments described above use a 2-dimensional grid pattern and a2-dimensional detector array 1. A similar arrangement can be used with a1-dimensional grid pattern, i.e. a single line of alternate light anddark areas, and a 1-dimensional detector array, i.e. a single line ofdetector pixels. Such an arrangement is illustrated in FIGS. 8A and B.FIG. 8A shows a linear detector array 1A, a lens system 2, lamp 5,projector lens 6 and grid 7A and a beam splitter 8 arranged in a similarmanner to the components of the embodiment shown in FIG. 1. An optionalscanning mirror 19 (to be described further below) is also shown. FIG.8B shows front views of the linear CCD detector array 1A and the lineargrid pattern 7A comprising a line of light and dark areas which havedimensions corresponding to those of the pixels of the CCD detector 1A.

The 1-dimensional arrangement shown in FIG. 8 corresponds to a singlerow of data from a 2-dimensional sensor and gives a 2-dimensionalcross-sectional measurement of the object 3 being viewed rather than afull 3-dimensional image. There are many applications where this is allthat is required but a full 3-dimensional image can be obtained if amechanism is used to scan the beam through an angle in order to give theextra dimension. In FIG. 8A, a scanning mirror 19 is shown performingthis task although many other methods could be used. The scanning device19 may either operate faster or slower than the sweep scan of the lenssystem 2 whichever is more convenient, i.e. either a complete lenssystem 2 focus sweep can be carried out for each scan position or acomplete scan of the object is carried out for each position of the lenssystem 2 sweep.

Alternatively, the extra dimension can be obtained by moving the objectbeing viewed. This arrangement is therefore suitable for situationswhere the object is being carried passed the sensor, e.g. by a conveyorbelt.

A 1-dimensional system has a number of advantages:

1) Lower cost when only 2-dimensional information is required.

2) The much lower data rates from the 1-dimensional detector allows morerapid sweep scans (the full 3-dimensional system described above islimited by the video data readout speed, i.e. 50-60 Hz for standardvideo). A 1-dimensional detector can be read out at many thousands ofscans/sec.

3) Less computing is required to process 2-dimensional data rather than3-dimensional data.

4) Longer detector arrays are available in 1-dimensional form. Typical2-dimensional detectors have about 512×512 pixels, whereas1-dimensional, linear detectors having 4096 pixels are readilyavailable.

5) Perhaps the most important advantage is that when the object beingviewed is not in focus, the signal level at the detector 1A is muchlower than for a 2-dimensional arrangement. This is because, theprojected light falls over a large area, most of which is not viewed bythe detector 1A. Hence, the further out of focus the object is, thesmaller the observed signal will be. This helps simplify the dataanalysis as the fall off in intensity is proportional to the"out-of-focus" distance (as well as the normal factors). This is incontrast to the 2-dimensional detector arrangement in which the averagelight level remains much the same, and simply falls off with thedistance between the sensor and the object according to the normalinverse square law.

A 1-dimensional version of the sensor is therefore suitable for use insensing:

1) Logs in a saw mill, e.g. in deciding the most economic way to cuteach log.

2) Extrusions, e.g. for checking the cross-sections of extruded metal,rubber or plastics articles or processed food products.

3) Food, e.g. for measuring the size of items of fruit, fish etc.

4) Objects carried by conveyors, e.g. for identifying objects or theirposition and orientation.

In another extension of the principle described above, the grid patternand detector of the sensor may each be further reduced to a `structuredpoint`. In practice, this can be provided by a small grid pattern, e.g.in the form of a two-by-two chequer-board pattern 7B and a smalldetector, e.g. in the form of a quadrant detector 1B, as shown in FIGS.9A and 9B. Alternatively, a two element detector and corresponding twoelement grid pattern may be used. The signals from such a detector areprocessed as if forming a single pixel of information. It will beappreciated that the optical arrangement of such a sensor is similar tothat of a scanning confocal microscope but with the addition of astructured light source and detector rather than a simple point.

In its simplest form, this form of sensor acts as a 1D range fin As inthe 1-dimensional detector case described above, extra dimensions can beobtained by using scanning mechanisms to deflect the beam across theobject of interest and/or by moving the object past the sensor, e.g. ona conveyor. A double galvanometer scanner could, for example, be used togive a full 3-dimensional image or a single galvanometer scanner to givea 2-dimensional cross-sectional image.

With temporal modulation the illuminated pair of grid elements isalternated between observations. However, the data rates possible usingpiezo-electric positioning means would be too slow to exploit thesampling rates of the detector so other means should preferably be usedto alternate the light source, such as:

1) a Faraday effect screen and projection optics.

2) a pair of matched laser diodes feeding light into pairs of fibreoptics which are connected to opposite pairs of the quadrant.

3) using four matched light sources, e.g. laser diodes, arranged in aquadrant and connected together in opposite pairs with an optical systemto focus the light onto a smaller quadrant grid mask.

4) using an acousto-optic modulator to deflect a laser beam between thesections of the grid pattern.

With this form of sensor, the mean illumination level remains constantand the AC synchronous component of the signal is detected to indicatewhen the image is in focus (as the opposite pairs of quadrants will bealternately bright and dark). The out of focus components will simplyprovide a level DC signal (as all four quadrants receive similar signalsfor both states of the illumination grid). The electronics is arrangedto detect the difference in signals received by opposite pairs ofquadrant in synchrony with the changing light source.

With this form of sensor, all the projected light has to come from thesmall, structured light source, so the most efficient arrangement is touse a laser as the light source. Also, available quadrant detectors maybe larger than required, so an intermediate imaging stage may berequired to match the size of detector with the grid pattern.

As with the 1-dimensional sensor described above, the intensity levelfalls off rapidly when the object is out of focus. In this case,however, the fall off is very much faster as (in addition to the normalfactors) the fall off is approximately proportional to the square of the"out-of-focus" distance (as similarly occurs in a scanning confocalmicroscope).

This provides the possibility of using the sensor for imaging indifficult situations, for example through fog, smoke or in cloudy water.This is because:

1) Each small area of the object is looked at in turn. (Illumination ofthe entire object would result in `fogging out` the entire image in thesame way as using full beam headlights of a car in fog can reducevisibility due to an increase in the amount of scattered light).

2) The intensity of the unwanted scattered light from out-of-focusregions of the object is kept to a minimum by the fall-off beingproportional to the square of the `out-of-focus` distance as describedabove.

3) The scattered light that does reach the detector will be out-of-focusand so effect all four quadrants equally. This contributes to `I` (themean signal) but contributes very little to the difference term `M`.

4) The synchronous detection scheme described above furtherdistinguishes spurious light scattered from the fog and any backgroundillumination from the genuine signal reflected from an in-focus object.

As with the 1-dimensional version described above, whether the lensfocus sweep is fast and the image scanning slow or vice-versa is amatter of convenience. With an intelligent system, a particular samplecan be terminated when the in-focus position has been found and it canthen proceed to the next position, using the current range informationas the starting guess for the next element, and so on.

FIG. 10 shows the overall block structure for the embodiment shown inFIG. 9 having a simple quadrant detector and grid pattern. This systemassumes that the quadrant light source is constructed using two matchedlaser diodes connected to pairs of fibre optics arranged in a quadrantpattern that matches the detector. A clock signal is used to synchronizethe activity of the light sources and detection electronics. The lightsources are illuminated alternately in such a way that the average lightintensity remains constant. The outputs from the four quadrants A,B,C,Dof the detector are amplified and combined to form the following terms.##EQU3##

This modulation depth signals M' is then passed through a standardsynchronous detection circuit. The signal M' is multiplied by plus orminus one depending upon the phase of the clock. The output M' and I'then pass through a low pass filter and a divider to give a signal Mthat corresponds to a measure of the `in-focus` signal. The signals Mand I are then digitized and sent to the computer.

Clearly the ordering of the above operations could be changed with nonet difference to the result. For example, the division could beperformed before or after the low pass filtering. The division couldalso be performed digitally after the analog to digital conversion (ADC)stage.

If necessary, the signal M may be integrated to give a better signal tonoise ratio. A test will also be required to reject measurements where Iis very low.

As discussed above, the characteristics of this sensor are such that themean intensity I rapidly drops away if the sweep position is not closeto the in-focus position. This fact can be used to skip rapidly overnon-useful data.

The net effect is that only the temporally modulated signalcorresponding to the in-focus grid pattern is detected. The lens sweepposition corresponding to the maximum M value can thus be determined andhence the range of the object. By combining this procedure with a beamscanning mechanism, 2-dimensional or 3-dimensional range maps can beconstructed.

The projection of a structured image onto the object which is to beviewed, together with the use of a detector able to sense the structureof the image, provides the advantages discussed above and enables theresultant image received by the detector 1B to be quickly and reliablyanalysed in the manner described to provide the required information. Inaddition, the use of this form of image and detector, enables a systemwith sufficient performance for a wide variety of applications to beprovided using relatively simple and inexpensive apparatus. In all thearrangements described, temporal modulation (in which the grid patternis moved half a repeat distance or the light and dark portions of theimage are alternated between observations) is used as previouslydescribed to simplify the analysis of the signals received by thedetector.

It will be appreciated that shifting the signal analysis into thetemporal domain provides significant advantages over prior arttechniques. The analysis is made a lot simpler so allowing the signalsto be analysed more quickly so a greater amount of image information canbe obtained within a given time. Individual pixels of the detector canalso be analysed independently of each other.

The performance of the sensors described above will depend criticallyupon the characteristics of the lens system 2. Depth resolution improvesproportionally to the aperture of the lens and the larger the apertureof the lens the better. Depth resolution also requires a lens of highresolving power. One way to improve resolution is to restrict thespectral bandwidth (to minimize chromatic aberration) with filters or bychoice of light source e.g. laser diodes. A filter would then be used infront of the detector 1 to match the light source in order to cut outunwanted background light.

The lens system 2 may simply comprise a fast, high quality camera lenswhich has the advantage of being relatively inexpensive.

A single element large aperture aspheric lens may also be used. This hasminimal spherical aberrations and would give good resolution if usedover a narrow spectra bandwidth.

A zoom lens may also be used. A wide angle view could then be taken,followed by a telephoto close-up of areas of interest to give a higherresolution. A disadvantage of most zoom lenses is their restrictedaperture which limits the depth resolution possible. The large number ofoptical surfaces within the lens system also tend to lead to problemswith stray light.

A mirror lens may be the preferred option for larger scale ranging asconventional lenses become too heavy and cumbersome. A camera mirrorlens is compact and its Cassegrain geometry offers a triangulationdiameter greater than its aperture number and so should provide betterdepth resolution.

A variable focus lens system may also be used. One problem with a normallens system is that the magnification changes during the sweep of thelens (as the detector to lens distance changes). This means that it isnot possible to associate a particular pixel with a direction in space,and the same pixel can come into focus more than once during the lenssweep (at least in principle, particularly for those pixels towards theedge of the detector). By using a variable focal length lens system suchas a zoom lens, we can in principle perform the sweep in focusingdistance and at the same time compensate by changing the focal lengthsuch that the net magnification, i.e. the field of view, remainsconstant. With this arrangement, it becomes possible to associate eachpixel with a direction vector in space which can simplify the analysistask of building the 3D model of the object.

Another alternative is to change the radii of curvature of the lens toadjust the focus. This can be done, for example, with a flexible liquidfilled lens, in which the internal pressure controls the curvature ofthe lens, in combination with a conventional lens. With this type oflens, the optical components remain in a fixed position during the sweepand the change in focus is achieved by changing the curvature and hencethe focal length of the lens. The range in lens power required for thesweep is relatively small so the liquid filled lens need only berelatively weak compared to the conventional lens which provides most ofthe power of the lens system. Other types of variable focus lens systemscould also be used.

Ideally the lens system should have a flat image plane, but in practicethat may not be the case. The circular symmetry of the lens systemshould, however, imply a similar symmetry to the image plane position. Acalibration procedure can be used to establish the relationship betweenthe lens sweep position and the in focus position as a function of theoff-axis distance. From this a calibration table can be constructedrelating the target distance to sweep position and off-axis distance.Suitable functions can then be fitted and used in convertingobservations into range. In addition, if a suitable calibration patternis used that can be read by the detector, e.g. a piece of graph paper,the scaling in the image plane can be automatically determined andfitted in a similar manner.

A possible problem with the optical sensors described above is thelikelihood of stray light from the light source entering the detector 1.Such stray light will not be in focus, but will have the effect ofreducing the contrast of the images and thus degrading performance. Thisproblem becomes worse as the scale of the ranging increases as a largerscale instrument will require a brighter light source and will receiveback less light from the object being viewed. In order to minimize thisproblem a number of strategies are possible:

1) Use a high quality lens system with multi-coated surfaces.

2) Match the field of view (FoV) of the light source to the needs of thelens system.

3) Use crossed polarizing filter over the grid pattern and the detector.

4) Use a polarizing beam-splitter with the above.

5) Use the minimum number of optical surfaces (a single element asphericlens may be useful).

6) Make sure the unwanted half of the beam that emerges from thebeam-splitter is well absorbed.

FIGS. 4 and 5 show another approach to the stray light problem in thecase of large scale ranging where the single lens is replaced by amatched pair of separate lens for the light source and detector. In FIG.4, the two lenses are shown mounted one above the other and with theiraxes parallel. In this case, a vertical grid pattern must be usedbecause of parallax and the lenses should have a flat image plane sothat the vertical bands align with the pixel array of the detector. Anintermediate solution is for the two beams to be combined externallyrather than internally as shown in FIG. 5. As less optical surfaces areinvolved, the stray light problem will be less acute. In this secondexample, the preferred chequerboard grid pattern can again be used.

The other unwanted source of illumination is background or ambientlight. This becomes progressively worse as the scale of the rangingincreases and will determine the upper limit of usefulness of thesensor. The "active" projected light source must therefore be dominant.A number of precautions can be applied to minimize this problem.

1) Use a bright, narrow spectra-bandwidth light source with matchingfilter over the detector.

2) Use a stroboscopic light source with synchronized short exposureframe-grab.

3) Capture images with no internal light source and subtract thisbackground signal from the active images.

The sensor relies on the triangulation angle afforded by the finitediameter of the lens system 2. Light is collected from the object 3 overthe full area of the lens system 2 and the larger this area the smallerthe depth of focus. In the same way, it is important that the outgoinglight from the sensor also makes full use of the total lens areaavailable in order to project a pattern with a short depth of field ontothe object 3. The optics of the light source should therefore bedesigned so that from each point on the grid a cone of light is directedtowards the lens so as to cover its full area. This cone of light shouldbe of a uniform intensity (or even perhaps biased towards the edge ofthe lens which contributes the greater triangulation angle).

Optical sensors of the type described herein have a wide range ofpossible applications. These include:

1) Autonomous guided vehicles (AGU's), i.e. free ranging roboticvehicles which use their own sensors to plan their motion.

2) Medical imaging, e.g. facial mapping for planning plastic surgery ormapping of a patient's back to investigate spinal disorders.

3) Dentistry, e.g. mapping dental cavities for automatic machining ofceramic filling inserts.

4) Inspection of products such as printed circuit boards, e.g. checkingcomponents are in place etc.

5) Industrial gauging, e.g. measuring components such as vehicle bodiesautomatically.

INDUSTRIAL APPLICABILITY

It will be appreciated from the above that the sensor described hereincan be used in a wide variety of industries and a wide variety ofapplications.

I claim:
 1. An optical sensor comprising:a structured light source for producing a pattern of contrasting areas; a detector which comprises an array of detector elements having dimensions matched to the pattern produced by the light source; an optical system for projecting a primary image of the light source onto an object that is to be sensed and for forming a secondary image on the detector of the primary image thus formed on the object; adjustment means for adjusting at least part of the optical system to vary the focussing of the primary image on the object, such that when the primary image is in focus on the object, the secondary image on the detector is also in focus; and processing means for analyzing signals produced by the detector in conjunction with information on the adjustment of the optical system; wherein the structured light source is adjustable to interchange at least two positions of contrasting areas of the pattern produced by the light source and in that the processing means is arranged to analyze the secondary images received by the detector elements with the contrasting areas in the interchanged positions to determine those parts of the secondary images which are in focus on the detector and thereby determine the range of corresponding parts of the object which are thus in focus.
 2. An optical sensor as claimed in claim 1 in which the structured light source is arranged to produce a pattern of light and dark areas and adjustment means are provided for moving the structured light source to interchange the positions of the light and dark areas.
 3. An optical sensor as claimed in claim 1 in which the structured light source comprises a plurality of elements arranged in a pattern which can be alternately switched on and off to interchange the positions of contrasting areas of the primary and secondary images formed on the object and on the detector.
 4. An optical sensor as claimed in claim 1 in which the structured light source comprises a regular pattern having a repeat distance which is n times the dimensions of the detector elements, where n is an even integer.
 5. An optical sensor as claimed in claim 1 in which the processing means is arranged to analyze signals representing two intensities of illumination received by each detector element when the image thereon is in the two respective positions and to determine those areas of the detector for which the difference between the two signals is relatively high and thus indicate the parts of the secondary image which are in focus on the detector.
 6. An optical sensor as claimed in claim 1 in which the detector comprises a 2-dimensional array of detector elements and the structured light source comprises a corresponding 2-dimensional pattern of contrasting areas and in which the processing means is arranged to form a 2-dimensional image of those parts of the object which are in focus for a given focussing adjustment of the optical system and to build up a 3-dimensional model of the object from a plurality of said 2-dimensional images as the focussing adjustment of the optical system is varied.
 7. An optical sensor as claimed in claim 1 in which the detector comprises a linear array of detector elements and the structured light source comprises a corresponding linear pattern of light and dark areas and in which the processing means is arranged to build up a 2-dimensional image of a cross-section of the object being sensed by detecting those parts of the secondary image which are in focus on the detector as the focussing adjustment of the optical system is varied.
 8. An optical sensor as claimed in claim 1 in which the detector comprises a structured point detector and the structured light source comprises a structured point source and in which the processing means is arranged to determine the range of an object being sensed by detecting when the structure of the image formed on the detector, as the focussing adjustment of the optical system is varied, corresponds to the pattern of the structured light source.
 9. An optical sensor as claimed in claim 7 including scanning means for scanning the image produced by the light source relative to the object being sensed whereby at least one additional dimension may be added to the image the sensor is able to form of the object.
 10. An optical sensor as claimed in claim 8 including scanning means for scanning the image produced by the light source relative to the object being sensed whereby at least one additional dimension may be added to the image the sensor is able to form of the object.
 11. An optical sensor as claimed in claim 2 in which the pattern of light and dark areas comprises light and dark bands.
 12. An optical sensor as claimed in claim 2 in which the pattern of light and dark areas comprises a checker-board pattern.
 13. An optical sensor as claimed in claim 4 in which n=2.
 14. An optical sensor as claimed in claim 8 in which the structured point detector is in the form of a quadrant sensor and the correspondingly structured point source is in the form of a two-by-two checker-board pattern.
 15. An optical sensor as claimed in claim 8 in which the structured point detector is in the form of a two element sensor and the correspondingly structured point source is in the form of a two element pattern.
 16. A method of determining the range of at least part of an object being viewed using an optical sensor comprising: a structured light source which produces a pattern of contrasting areas; a detector which comprises an array of detector elements having dimensions matched to the pattern produced by the light source; an optical system which projects a primary image of the light source onto an object that is to be sensed and forms a secondary image on the detector of the primary image thus formed on the object; adjustment means which adjusts at least part of the optical system to vary the focussing of the primary image on the object such that when the primary image is in focus on the object, the secondary image on the detector is also in focus; and processing means which analyzes signals produced by the detector in conjunction with information on the adjustment of the optical system; said method including the steps of:(a) adjusting the structured light source to interchange the positions of contrasting areas of the pattern produced by the light source; (b) analyzing the secondary images received by the detector elements with the contrasting areas in the interchanged positions to determine those parts of the secondary images which are in focus on the detector; and (c) determining the range of corresponding parts of the object which are thus in focus. 